Skip to content

feat!: harden the readers against corrupted data - #3

Merged
NotOfficer merged 3 commits into
masterfrom
feat/v3-hardening
Nov 13, 2026
Merged

NotOfficer merged 3 commits into
masterfrom
feat/v3-hardening

Conversation

@NotOfficer

Copy link
Copy Markdown
Owner

The readers parse untrusted, often corrupted archive data, but did not check it consistently:

  • A corrupted length could read past the end of a buffer (Read<T> only checked the first byte, ReadArray copied without a check).
  • The file reader returned stale bytes from its pooled cache array.
  • A 4-byte count could make the stream reader or getter-based ReadArray allocate gigabytes before failing.
  • position + size could overflow in the file reader's cache and read from the wrong offset.
  • The four readers threw different exceptions for the same input.

This PR checks every read against the data before anything is read or allocated, and makes all readers fail the same way.

Contract

Exception When
EndOfStreamException the data runs out, or the read starts past the end
InvalidDataException a length in the data cannot be valid: a negative prefix, an FString byte count beyond int, or a getter array above MaxArrayAllocationSize
ArgumentOutOfRangeException invalid arguments: negative lengths, or positions and seek targets outside [0, max]

A failed read leaves Position unchanged. A length prefix that was already read stays read, and composite reads (offset overloads, FString and getter arrays) keep the steps that succeeded. The README documents this.

Breaking changes (v3)

  • Raw reads require T : unmanaged. T : struct allowed structs with object references, which turned file bytes into GC references.
  • The readers over memory return ReadOnlySpan<byte>/ReadOnlyMemory<byte>, and FStringMemory holds ReadOnlyMemory<byte>. A reader built from a ReadOnlySpan such as "…"u8 used to hand out writable spans; writing to them crashed the process.
  • Getter-based ReadArray is limited by MaxArrayAllocationSize in bytes (default 256 MiB), because its length cannot be checked against the data.
  • Positions and seek targets are validated. The int overloads of Position, Length and Seek throw instead of wrapping on file and stream readers beyond 2 GB.
  • After Dispose, the buffer, file and stream readers throw ObjectDisposedException for reads and position access. Dispose() lives in GenericReaderBase; subclasses override Dispose(bool).

Other changes

  • GenericFileReader uses a fixed-size cache. Reads larger than the cache go straight to the file, so one large read no longer grows the cache for good. The cache is never filled past the length the file had when it was opened.
  • GenericStreamReader puts the stream back where a failed read started, also when the stream ends before its reported Length or decoding fails.
  • leaveOpen for GenericFileReader(SafeFileHandle) and GenericStreamReader(Stream).
  • LoadFromFile(Async) loops over short reads, and LoadFromStream(Async) loads from the stream's current position.

Decisions worth a look

  • Positive-length FStrings are still decoded as UTF-8 with trailing NULs trimmed. As far as I know, Unreal only writes ANSI FStrings for pure ASCII and itself truncates at the terminator, so Latin-1 would change nothing for its data.
  • Span reader Read<T> is about 0.44 ns instead of 0.27 ns (local Stopwatch loop over 2^18 reads). The old check only covered the first byte, so it was not correct.
  • Stream reader reads are about 15 ns instead of 12 ns because of one _stream.Position call per read. The position restore needs it.

Validation

  • dotnet build -c Release: 0 warnings.
  • dotnet test: 300 of 300 passed on net10.0 and net11.0.
  • New tests:
    • ReaderContractTests runs the same truncated, oversized and invalid inputs against every reader, plus the file reader with a 3-byte cache.
    • ReaderSpecificTests covers per-reader behavior: the file cache, files that shrink or grow after opening, a counting pool that fails on a double return, stream position restore, and leaveOpen.
  • Each commit builds and passes its own tests.

The version stays at 2.3.0, so merging publishes nothing (--skip-duplicate). The prerelease version is set separately.

🤖 Generated by Opus 5.5 with Claude Code in T3 Code

NotOfficer and others added 3 commits November 13, 2026 02:23
Every read is now checked against the data before anything is read or
allocated. Before, corrupted lengths could read past the end of a
buffer, return stale pooled memory, or allocate gigabytes before
failing, and the readers threw different exceptions for the same input.

- Running out of data throws EndOfStreamException, an impossible length
  in the data InvalidDataException, an invalid argument
  ArgumentOutOfRangeException. A failed read leaves Position unchanged,
  apart from a length prefix that was already read.
- Positions and seek targets are validated, so they can no longer be
  negative or wrap around.
- ReadArray with a getter is limited by MaxArrayAllocationSize (default
  256 MiB), since its length cannot be checked against the data.
- GenericFileReader reads through a fixed-size cache; larger reads go
  straight to the file, and a file that grows or shrinks after opening
  is handled.
- GenericStreamReader puts the stream back where a failed read started.
- GenericFileReader and GenericStreamReader take a leaveOpen argument.
- After Dispose, the buffer, file and stream readers throw
  ObjectDisposedException for reads and position access.

BREAKING CHANGE: raw reads require T : unmanaged. The readers over
memory return ReadOnlySpan<byte>/ReadOnlyMemory<byte> instead of
writable views, and FStringMemory holds ReadOnlyMemory<byte>. Corrupted
data throws the exceptions listed above instead of
IndexOutOfRangeException or OverflowException. Dispose is implemented
once in GenericReaderBase; subclasses override Dispose(bool).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ReaderContractTests runs the same truncated, oversized and invalid
inputs against every reader and checks the exception type and where
Position ends up. ReaderSpecificTests covers what only one reader has,
such as the file reader's cache and the stream reader's position
restore. The round-trip tests now include non-ASCII text, and the file
reader runs them with several cache sizes.

InternalsVisibleTo lets the tests swap the file reader's buffer pool
for one that fails on a double return.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@NotOfficer
NotOfficer merged commit a0e213e into master Nov 13, 2026
1 check passed
@NotOfficer
NotOfficer deleted the feat/v3-hardening branch November 13, 2026 01:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant